Repository navigation
Conversation
Duel (https://github.com/sferik/duel-ruby) runs the benchmarks in duel/ on the branch and on master side by side, and fails a pull request that is slower or allocates more. The 44 benchmarks cover queries over a socket to a PostgreSQL server of each worker's own (exec, exec_params, exec_prepared, COPY in and out), reading results in memory (values, getvalue, each, each_row, tuples, field names), the type maps, the text and binary coders on their own, escaping, and parsing connection parameters. Each version compiles its own extension (c.build). The duel-postgresql plugin needs a released pg, so Duel is in an optional Gemfile group, and .duel.rb puts each version's lib/ first and the release off the load path before the plugin requires pg. Checked by comparing master with itself three times, and with planted regressions: the Integer decoder without its fast path failed 6 benchmarks on instructions, and an extra rb_str_dup in the String decoder failed 13 on allocations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in
duel/againstmasterand the branch side by side, in alternating worker processes, and fails the build if the branch is slower or allocates more. It judges CPU time and instruction counts with confidence intervals, and counts allocations exactly. When something regresses, it profiles both versions with StackProf and says where the extra time went.There are 44 benchmarks, and most of them stay away from the server on purpose, so that what's timed is the driver and not PostgreSQL. Results are fetched once and then read in memory with
values,getvalue,each(with string and symbol keys),each_row,tuple,column_valuesand the field name and type lookups. The type maps decode the same 100-row result throughBasicTypeMapForResultsin text and binary format and through aTypeMapByColumn, andBasicTypeMapForQueriespicks encoders for a mixed set of parameters and feedsembed_params. The coders get benchmarks of their own, 100 fixed values a call: the Integer, Float, Date and both Timestamp text decoders (with the recent work on those in mind), integer and quoted-string arrays, Bytea, the matching encoders and the binary Int8, Float8 and TimestampUtc coders. Then there'sescape_string,escape_literal,quote_identand the bytea round trip, the CopyRow encoder and decoder, andparse_connect_argswith a URL, a keyword string and a Hash. The rest do go to a server:exec,exec_params,exec_prepared, 100 rows with and without a result type map, and COPY in and out with coders. They use the duel-postgresql plugin, which starts a PostgreSQL server for each worker on a Unix socket and also judges each benchmark on the queries and rows the server ran.The tricky part was pg being the plugin's own dependency. duel-postgresql needs a released pg, and it used to require it as
.duel.rbloaded, before Duel loaded the version. So every worker was quietly measuring pg 1.7.0 from the bundle instead of either version. I checked this with a probe that wrote out each worker's$LOADED_FEATURES. I fixed that in duel-postgresql 0.1.4, which requires pg only once the worker's version has loaded, so.duel.rbneeds no workaround. With it, the probe showed each worker with its ownpg.rbandpg_ext.bundleand nothing else. The released pg also brokerake speconce it was in the bundle, because its fat binary gem has alib/3.4/pg_extthatpg.rbtries first. So Duel lives in an optionalduelgroup in the Gemfile, with the released pg as the source gem, andbundle installand the specs are exactly as they were unless you opt in. With the group installed the specs still pass (1163 examples, 0 failures).A full run takes 20 to 25 seconds on my laptop, and the Duel job took about two and a half minutes on GitHub Actions, most of it setup and compiling both versions.
The Duel job only reads. The comment on a pull request comes from a second workflow,
perf-comment.yml, which runs none of the pull request's code and shows a report from a fork as plain text. If a change is slower on purpose, aduel: acceptlabel lets it through.